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Integrated learning systems (ILSs) provide a multi-year curriculum sequence of computer-based instruction 
controlled by a sophisticated management system . ILS implementation is better understood when the focus of 
implementation is shifted from the technology to the people who use the technology . This study examined the 
operational patterns of teachers implementing an ILS in elementary schools in a metropolitan school district. A 
measure was developed that described the major components of imple7 mentation of ILS technology. Data regarding 
the operational patterns of teachers using ILS technology were collected from teacher interviews and analyzed . The 
best ILS implementation practices included integration with classroom instruction, training in the use of an ILS, and 
the use of motivational strategies. Implementation is better understood when contextual phenomena of the 
implementation are examined because the question of the instructional effectiveness of an ILS or other forms of 
computer-delivered instruction may be as much an implementation issue as a matter of instructional design. 

Introduction and Background of Study 



To suppose that users of computer-delivered instruction implement courseware in the way it is intended to 
be used is often an ambitious assumption. Smith and Ragan (1993) advised that "in drawing the line of causation 
from the instruction to the results, it is critical to be able to identify the degree to which the description of the 
program represents what actually occurred during instruction with the new program" (p. 416). Therefore, the 
determination of effectiveness of computer-delivered instruction needs to begin with a determination of the degree of 
implementation of the instruction. 

This article describes the development, validation, and research application of an instrument for evaluating 
the implementation of computer-delivered instruction. This instrument, the Computer-Delivered Instruction 
Configuration Matrix (CDICM), was designed to examine the implementation of Integrated Learning Systems 
(ILSs). The instrument was validated in a study by Mills (1997) involving four metropolitan elementary schools. The 
purpose of this paper is to describe the development and deployment of the CDICM and its implications for 
educational change and reform. 

The CDICM was adapted from the Innovation Configuration Matrix (ICM), a tool developed by Heck, 
Steigelbauer, Hall, and Loucks (1981) for use in identifying the essential components of an innovation and the 
variations of implementation for each of the innovation components. The ICM is an effective resource for assessing 
implementation of instructional technology and other educational innovations (Albers, 1994; George & Hord, 1980; 
Gleghom, 1993; Hord, Huling, & Austin, 1986). The ICM provides an understanding of the operational patterns of 
users of an innovation and therefore allows the evaluators of an innovation to make judgments about appropriate 
practices and how much variation in practices is acceptable (Hord, Rutherford, Huling- Austin & Hall, 1987). 

Hall and Loucks (1977) were disappointed when experimental research and evaluation studies presumed the 
treatment was present and its effects were accounted for by testing for statistically significant differences between 
two groups or by pre- and post-testing the same sample. They concluded that information about the actual use and 
degree of implementation of an innovation might better explain some nonsignificant findings reported in evaluation 
and experimental studies. According to Hall and Loucks (1978) a particular innovation can have several different 
operational patterns as a result of variations in the selection and use of innovation components. They labeled these 
operational patterns as innovation configurations. Different patterns or configurations resulted when different 
teachers put innovations into operation in their classrooms. Individual teachers used different components of an 
innovation in different ways and when these components were put together a number of patterns emerged that 
characterized different uses of the innovation (Hord, Rutherford, Huling-Austin, & Hall, 1987). 
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The construction of the CDICM was based on a review of research on computer-delivered instruction, 
courseware documentation, telephone interviews with courseware developers, and interviews with vendor training 
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facilitators, lab managers, and teachers who used the Computer Curriculum Corporation (CCC) courseware, 
Successmaker' Successmaker courseware covers subjects ranging from madfand reading to GED preparation. 

: -Using ICM development techniques an ICM was developed to fit with the adoption and use of the 
Successmaker ILS in varying degrees and patterns of implementation. The matrix consisted of a component checklist 
comprised of five variations for each component with each successive variation indicating a level of use representing 
a closer approximation of ideal use. Although the CDICM instrument was developed and tested with a particular 
ILS, its application is relevant for many forms of computer-delivered instruction. 

The CDICM was devised as an instrument to be completed by an evaluator based on responses supplied by 
teachers during an interview. The components of technology use and variations of the components selected for 
inclusion on the CDICM were presumed to reflect the actual and ideal practices of teachers involved with ILS 
implementation. Although many forms and variations of technology adoption and use by teachers may exist, the 
process of instrument development attempted to reflect a true and accurate gradient of technology use from "least 
acceptable" to "ideal." Since truth and accuracy of instrumentation was not absolute, the intent was to develop an 
instrument that provided estimates that possessed known and reasonable validity and reliability. 

As part of the development process, the CDICM was subjected to a two-phase field test. In the fust phase 
interviews were conducted with a sample of ten teachers who regularly used the courseware for instruction. The 
purpose of this administration of the CDICM was to make revisions to matrix components and to make revisions to 
the interview guide. After revisions were made, the second phase of the field test was conducted to establish 
reliability and validity for the final version of the CDICM. A sample of nineteen teachers who regularly used the 
courseware completed a CDICM checklist using the revised CDICM and then five of the subjects completing the 
checklist were selected for interviews. 

Descriptive statistics were computed for total instrument reliability. The final 15-item checklist allowed for 

a range of total scores from 0 to 75. The standard error of measurement for the field test was 3.68 while the 

interscorer reliability coefficient was .96 for all items. This administration of the CDICM checklist yielded a 
coefficient alpha of .61 (p < .01) for the total scale. The final version of the CDICM checklist and interview guide is 
provided at the end of this article. 

Data Collection and Analysis Procedures 

The final version of the CDICM was deployed in a school district that had made a substantial investment in 
ILS technology. ILS technology was used in all elementary school campuses in the district. The school district 
initially implemented ILS technology in six elementary schools and during the previous school year expanded the 
program to include almost all kindergarten through eighth grade students in the school district. 

A lab configuration was employed at each school with a number of workstations distributed to some 

classrooms. Each school had a lab coordinator or media specialist to coordinate the computer lab activities and to 

assist the classroom teacher in the management and coordination responsibilities of the computer lab and in 
facilitating ILS instruction. The school principal developed a lab schedule in which all students at each school were 
provided forty-five minute sessions using the ILS two or three times per week to supplement regular classroom 
instruction. 

The sample selected for this study consisted of elementary school teachers at four schools whose students 
interacted with an ILS using the Successmaker courseware for instruction. A sample of teachers (N=30) was 
randomly selected for the study from all of the ILS-using teachers in the four schools. The schools selected for this 
study had employed ILS technology for different periods of time and since the sample size was relatively small, no 
comparisons were made among the various schools. 

To collect data regarding patterns of use occurring among individuals implementing an ILS and differences 
among these patterns of use, a structured interview was conducted and recorded by the researcher with each of the 
teachers selected for the random sample. One interview in the original selection was not completed making N=29 for 
the CDICM. 

An expert group consisting of the researcher, an experienced lab manager (not from the school system under 
study), and a vendor representative was assembled to rate the interviews for the CDICM checklist. The CDICM 
component scores for each case in the sample were based on the collective ratings of the expert group. The expert 
group reviewed transcribed audio tapes of teachers' interviews and independently scored each respondent based on 
the content and structure of the CDICM checklist. 



■ Analysis of use by teachers implementing the ILS was conductel^tfirdugh pattern analysis of the interview 
data to determine dominant configuration patterns or variations. The general patterns that emerged were analyzed by 
a cluster analysis that identified relatively homogenous groups of cases based on selected characteristics. 

One-way analysis of variance was used to examine differences among the configuration patterns for each of 
the components measured by the CDICM. A test was performed to validate the results of the ANOVA against 
discrepancies that existed in the assumptions due to differences in group sizes and normality (Minium, 1978; 
Norusis, 1995). Post-hoc tests were performed to compare differences between pairs of configuration patterns 1 
means for each CDICM component using the Bonferroni procedure. 

An independent variable was constructed for CDICM level of use by assigning a level of use classification 
for the composite score of the CDICM to each case (using the same scale as was used for the component variations 
on the CDICM (1, 2, 3, 4 or 5). Next, a linear regression analysis was used to examine the relationship between each 
significant CDICM component (from the ANOVA) and the degree of ILS implementation. A stepwise multiple 
regression was performed to determine which components were the best predictors of ILS implementation. 

Results of the Data Analysis 

Reliability statistics were computed to determine the internal consistency in the ratings determined by each 
scorer as well as the external consistency in ratings among the scorers. External consistency of the ratings among the 
three scorers for all items on the scale was r = .9926 for AB, r = .9974 for AC, and r - .9935 for BC. Internal 
consistency was indicated by a significant coefficient alpha (p < .01) for each scorer and for all scorers combined. 

Pattern analysis of the interview data was performed to determine dominant configuration patterns or 
variations. This procedure identified relatively homogenous groups of cases by computing the centroid for each 
dominant configuration pattern that emerged from the data and then assigning each case to the cluster with the 
nearest centroid until no cases changed cluster membership. Since the initial cluster centers and the number of 
dominant patterns were unknown, this procedure consisted of selecting all fifteen components of the CDICM for use 
in the cluster analysis and incrementing the number of clusters until a reasonable model was obtained. The number of 
clusters started with two and was incremented by one until convergence of cluster centers reached a distance of zero 
and the number of cases in each cluster was similar. The cluster analysis was run for two, three, and four clusters 
before a reasonable model was selected. 

At two clusters convergence was achieved with a maximum distance of center change of .3205 while at 
three and four clusters convergence was achieved with a maximum distance of center change of .0000. At two 
clusters the distance between final clusters centers was 7.1883 with 18 cases in one cluster and 1 1 cases in the other. 
However, at four clusters the distances between final cluster centers ranged from 7.6997 to 17.1270 and the number 
of cases in each cluster ranged from 1 to 16. The best model occurred with the number of clusters set at three. When 
the number of clusters was set at three, convergence of cluster centers was achieved after four iterations and the 
maximum distance by which any cluster center changed was zero. When the number of clusters was set at three, the 
number of cases in each cluster ranged from 8 to 12. 

In order to assess the adequacy of the classification of implementation pattern groups by the cluster analysis 
a Discriminant Analysis (DA) was performed. The fifteen CDICM components were used to separate the groups into 
the discriminant functions. As a result of this procedure all grouped cases were correctly classified. 

Table 1 lists each CDICM component by F( 2,26) and the corresponding discriminant function coefficients 
and Figure 1 provides a plot of the discriminant functions for each of the CDICM patterns. The discriminant function 
coefficients reflected the importance attached to each CDICM component in distinguishing the discriminant 
functions. The multivariate analysis of variance performed by the DA produced significant differences among the 
Implementation Patterns on CDICM components 1, 2, 3, 4, 5, 11, and 12 (p < .05). Based on both functions of the 
DA, Pattern 2 was indicative of a high degree of implementation characterized by high training and support, 
integration into classroom and curriculum, and use of reinforcement and motivational strategies. 

To examine differences among the CDICM configuration patterns a one-way analysis of variance 
(ANOVA) was performed. One-way analysis of variance determined significant differences among the 
implementation patterns groups for each of the CDICM components. A test of homogeneity of variance was 
performed to validate assumptions required by the ANOVA and to determine whether the variance of the dependent 
variables was significantly different among the implementation pattern groups. For the most part, the assumption of 
equality of variances was satisfied. However, the Levene statistic produced by the homogeneity of variance test was 
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Table L Discriminant Function Weights for CDICM Components 



CDICM Component 


F 


_Sjgi 


Fn 1 


Fn 2 


12. Uses Reinforcement/Motivational Strategies 


67.8826 


.0000 


1.25823 


-.30846 


2. Received Training in Use of ILS 


10.2191 


.0005 


.13897 


.29295 


5. Integrates with Classroom Instruction 


9.2215 


.0009 


-.27445 


1.74723 


3. On-Going Support is Provided 


7.6386 


.0025 


.60909 


.18948 


1 L Facilitates ILS Instruction 


6.5705 


.0049 


.33663 


.14500 


1. Understands ILS Instructional Design 


3.4095 


.0484 


-.16839 


.28902 


4. Sets Instructional Goals for ILS 


2.8557 


.0757 


-.23317 


.61968 


13. Uses ILS Management Reports 


2.5546 


.0971 


-.36072 


-.09096 


10. Provides Sufficient Time on Task 


2.1375 


.1382 


1.48069 


-.53613 


9. Sets Clear Rules of ILS Use 


1.7029 


.2019 


-.15515 


.10001 


6. Integrates with District Curriculum 


1.5184 


.2379 


.35935 


-.59012 


15. Understands/Uses ILS Routines and Equipment 


.9081 


.4157 


-.07324 


.08089 


7. Individualizes Enrollment Options on ILS 


.6088 


.5516 


.25277 


-.42373 


8. Effective Scheduling of ILS 


.4905 


.6179 


-1.07805 


-.77891 


14. Uses ILS Achievement in Learner Evaluation 


.3634 


.6988 


.14913 


-.58905 



Canonical Discriminant Functions 




V Group Centroids 
A Group 3 
° Group 2 
° Group 1 



Function 1 



Figure 1. Plot of Discriminant Functions for Each CDICM Implementation Pattern. 



significant (p < .05) for two CDICM components: Component 7 — Individualizes Enrollment Options on the ILS and 
Component 9 — Sets Clear Rules of ILS Use. 

A plot of these two variables revealed that the homogeneity of variance assumption was violated by four 
outlier cases for Component 7 and three outlier cases on Component 9. Several attempts to transform these variables 
yielded no success in obtaining a nonsignificant Levene statistic and in satisfying the assumption of equality of 
variances and so these variables were discarded from further analysis. 



An ANOVA examined the differences among the implementation-patterns for each of the components 
measured by the CDICM (see Table 2). The observed power was calculated for'the Ftest (p< 05). The results of the 
ANOV'A were consistent with the multivariate analysis of variance performed by the discriminant analysis. The 
ANOVA indicated significant differences for six of the CDICM components. 

To validate the results of the ANOVA in regard to violations of assumptions about differences in group sizes 
and normality, a Kruskal-Wallis nonparametric analysis of variance was performed. The Kruskal-Wallis test 
confirmed the results of the ANOVA for five of the six significant CDICM components. Only Component 1 — 
Understands ILS Instructional Design yielded a significant difference on the ANOVA but not on the Kruskal-Wallis. 

Table 2. ANOVA of CDICM Interview Data Grouped by Implementation Pattern 



Dependent Variable 


df 


F 


Sig. 


Observed 

Power 


1 . Understands ILS 


Between Groups 


2 


3.409 


.048 


.589 


Instructional Design 


Within Groups 


26 










Total 


28 








2. Received Training 


Between Groups 


2 


10.219 


.001 


.975 


in Use of ILS 


Within Groups 


26 










Total 


28 








3. On-Going Support 


Between Groups 


2 


7.639 


.002 


.920 


Is Provided 


Within Groups 


26 










Total 


28 








4. Sets Instructional 


Between Groups 


2 


2.856 


.076 


.511 


Goals for ILS 


Within Groups 


26 










Total 


28 








5. Integrates with 


Between Groups 


2 


9.221 


.001 


.961 


Classroom Instruction 


Within Groups 


26 










Total 


28 








6. Integrates with 


Between Groups 


2 


1.518 


.238 


.293 


District Curriculum 


Within Groups 


26 










Total 


28 








8. Effective Scheduling 


Between Groups 


2 


.491 


.618 


.122 


of ILS 


Within Groups 


26 










Total 


28 








10. Provides Sufficient 


Between Groups 


2 


2.137 


.138 


.398 


Time on Task 


Within Groups 


26 










Total 


28 








1 1. Facilitates ILS 


Between Groups 


2 


6.570 


.005 


.874 


Instruction 


Within Groups 


26 










Total 


28 








12. Uses Reinforcement/ 


Between Groups 


2 


67.883 


.000 


1.000 


Motivational Strategies 


Within Groups 


26 










Total 


28 








13. Uses ILS 


Between Groups 


2 


2.555 


.097 


.465 


Management Reports 


Within Groups 


26 










Total 


28 








14. Uses ILS Achievement 


Between Groups 


2 


.363 


.699 


.102 


in Learner Evaluation 


Within Groups 


26 










Total 


28 








15. Understands/Uses ELS 


Between Groups 


2 


.908 


.416 


.190 


Routines and Equipment 


Within Groups 


26 










Total 


28 
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Post-hoc Bonferroni tests were used to compare differences between. pairs of configuration patterns' means 
for CDICM components yielding a significant difference on both the ANO'VA and the Kruskal-Wallis (see Table 3). 
The Bonferroni comparisons indicated significant mean differences (p < .05) for all five CDICM components for 
configuration Pattern 2 when compared to Patterns 1 and 3 except for Component 12. These post-hoc findings 
indicated that differences among the configuration patterns revealed by analysis of variance were primarily based on 
differences between Pattern 2 and the other two patterns. Post-hoc testing validated that these differences were 
significant for Components 2, 3, 5, 11, and 12. Thus, teachers who implemented the ILS according to Pattern 2 were 
statistically better implementers than teachers who implemented the ILS according to Patterns 1 or 3. 

Table 3. CDICM Components Significant at 95% Confidence Interval in Post Hoc Comparisons 



Dependent Variable 


Implementation 

Pattern 


Implementation 

Pattern 


Mean 

Difference 


Std. 

Error 


Si& 


2. Received 


1 


2 


-3.28* 


1.089 


.017 


Training in 




3 


1.31 


.988 


.593 


Use of ILS 


2 


1 


3.28* 


1.089 


.017 






3 


4.58* 


1.022 


.000 




3 


1 


-1.31 


.988 


.593 






2 


-4.58* 


1.022 


.000 


3. On-Going 


1 


2 


-2.39* 


.819 


.022 


Support Is 






.53 


.743 


1.000 


Provided 


2 


1 


2.39* 


.819 


.022 






3 


2.92* 


.769 


.002 




3 


1 


-.53 


. .743 


1.000 






2 


-2.92* 


.769 


.002 


5. Integrates with 


1 


2 


-4.26* 


1.084 


.002 


Classroom 




3 


-.56 


.984 


1.000 


Instruction 


2 


1 


4.26* 


1.084 


.002 






3 


3.71* 


1.018 


.004 




3 


1 


.56 


.984 


1.000 






2 


-3.71* 


1.018 


.004 


1 1. Facilitates ILS 


1 


2 


- 1.76* 


.608 


.022 


Instruction 




3 


.19 


.552 


1.000 




2 


1 


1.76* 


.608 


.022 






3 


1.96* 


.571 


.006 




3 


1 


-.19 


.552 


1.000 






2 


-1.96* 


.571 


.006 


. 12. Uses 


1 


2 


-7.64* 


.769 


.000 


Reinforcement/ 




3 


- 7.22* 


.698 


.000 


Motivational 


2 


1 


7.64* 


.769 


.000 


Strategies 




3 


.42 


.722 


1.000 




3 


1 


7.22* 


.698 


.000 






2 


-.42 


.722 


1. 000 



*The mean difference is significant at the .05 level. 



A stepwise multiple regression was performed to determine significant CDICM components or 
combinations of components that were the best predictors of the degree of ILS implementation (see Table 4). 
Stepwise variable selection entered each CDICM component into the regression model by order of importance and 
removed components with diminished importance as additional predictors were added. 

All CDICM components were entered into the regression model as predictor variables and the degree of 
ILS implementation as measured by CDICM composite score was entered into the regression model as the criterion 
variable. Significant correlations (p < .01) were obtained among all the variables. Component 5 — Integrates with 
Classroom Instruction explained 56% of the variability in the degree of ILS implementation for this model and 54% 



for. other-' data sets from the same population. Component 5, Component 2 -^Repeived Training in Use of ILS , and 
Component‘12 — Uses Reinforcement/Motivation Strategies explained 87% of the variability of the degree of ILS 
implementation for this model and 95% of the variability of the degree of ILS implementation was explained by 
Components 5, 2, 12, 4, and 6. 

Table 4. Stepwise Multiple Regression of Influence Variables on Degree of ILS Implementation 



Variable a,b 


R 


R 2 


Adjusted 

R 2 


Std. Error 
of Estimate 


5. Integrates with Classroom Instruction 


.748 


.560 


.544 


9.39 


2. Received Training in Use of ILS 


.855 


.732 


.711 


7.47 


12. Uses Reinforcement/Motivational Strategies 


.933 


.871 


.856 


5.28 


4. Sets Instructional Goals for ILS 


.967 


.935 


.924 


3.82 


6. Integrates with District Curriculum 


.974 


.950 


.939 


3.44 


15. Understands/Uses ILS Routines and Equipment 


.980 


.961 


.950 


3.11 


8. Effective Scheduling of ILS 


.984 


.969 


.959 


2.82 


14. Uses ILS Achievement in Learner Evaluation 


.989 


.978 


.969 


2.44 


11. Facilitates ILS Instruction 


.993 


.986 


.980 


1.98 


9. Sets Clear Rules of ILS Use c 


.995 


.990 


.984 


1.77 



dependent Variable is Degree of Implementation 

b Stepwise Method, Probability-of-F-to-Enter <= .050, Probability-of-F-to-Remove >= .100 
Probability of F-to-enter = .050 limits reached 



Implications for Educational Change 

Current debate in the field of educational change has concentrated on contrasting differences in reductionist 
and holistic orientations to scientific inquiry (see Banathy, 1995). Carr (1996) explained this debate by suggesting 
that integration and separation represented two fundamentally different aspects of the same reality. Integration was 
concerned that when a system was separated or reduced to smaller components some of its vital properties were lost 
while separation was concerned that the whole was too complex to be studied or understood in its entirety. 

The problem with the approach of the first decade of computer use in schools was that it was focused on 
applying the computer to the existing instructional delivery system. Collins (1991) described American schools as a 
self-sustaining, interlocking system of institutions consisting of age-graded schools, multipie-choice testing, 
curriculum and materials, teacher education, and lecture and recitation that naturally resisted technology: 

If you try to introduce computers for students to do their work, the change will be sustained only to the 
degree that it fits the prevailing institutional structure. Since computers undermine the lecture and recitation 
methods of teaching and promote the student as self-directed learner, they do not fit this institutional 
structure and will be squeezed out by it. (p. 32) 

However, for systemic change to occur the focus of change must become our view of the role of the teacher 
in implementing technology. Many researchers in the field of educational change are applying the principles of 
holistic thinking to the creation of new systems of human learning through methodologies that focus less on end 
goals and outcomes and more on helping individuals change their perceptions of themselves (Carr, 1996). The 
methodology followed by this study — emphasizing ILS implementation factors and focusing on users of ILS 
technology — and the subsequent data collection and findings support a systemic change process in education. 

The findings of this study provided evidence for the proposition that not all ELS use is the same. Significant 
differences and variances in both the ELS implementation concerns and behaviors of teachers implementing an ILS 
were noted. The level of ILS implementation was a function of implementation practices that included integration 
with classroom instruction, training in the use of an ELS, and the use of motivational strategies. 

Furthermore, using an ELS to improve the teaching/learning process was more complex than earlier 
understood. Just because a teacher was an effective unplementer of an ELS was no guarantee that learners realized 
higher achievement using the ELS. Without the necessary organizational support the expectation for instructional 
technology to improve the teaching and learning process cannot be sustained. 

This examination of the practices of teachers implementing an ELS can be a highly effective guide to actions 
that change agents and stakeholders might follow as they assume a role in facilitating change and improvement in the 



implementation of ILS technology. Given the context of ILS implementatio^ythe findings of this study substantiated 
the research on change and innovation. Several lessons from the change literature were revealed through this study 
that might assist stakeholders or change agents in promoting a successful and effective implementation of computer- 
delivered instruction: 

1. Implementation of an ILS is a developmental process that must be nurtured and sustained. Implementation of an 
ILS doesn’t happen by itself. When left to their own devices, teachers will implement an ILS to whatever level is 
consistent with their concerns about the ILS and structurally fits into their existing teaching patterns and 
practices. 

2. Teachers implement an ILS in different ways. The specific operational components of the ILS must be 
communicated to teachers so they understand what the program looks like when it is fully functioning. Before 
learner achievement data can accurately be examined to determine the effectiveness of the ILS, it must be 
determined to what degree the ILS has been implemented. A determination of the relative merit of an ILS should 
be based on an examination of the degree to which an ILS is actually used in relation to the intended use of the 
ILS. 

3. Integrate ILS instruction with classroom activities and instruction. For systemic change to occur in the 
teaching/leaming process, the existing system has to be fundamentally replaced with an improved system. 
Teachers that were the most effective implementers of an ILS incorporated ILS instruction into classroom 
instruction or vice versa. 

4. Training should be a continuous process and not a one-time event. On-going training is a key component to 
sustaining an ILS implementation. On-going training reaffirms fundamental practices that focus the user on the 
intended use of the ILS and influences the concerns a user will have about the implementation of an ILS. 

5. ILS implementation must be teacher-driven. There is a perception that an ILS runs itself (Gleghom, 1993) and 
by simply placing learners with computers does not ensure that they will grasp the underlying structure of 
important ideas and concepts. Resnick and Johnson (1988) noted that learning that occurs in isolation is not 
sustained in other contexts and Pea (1988) explained the necessity of teachers to assist learners in applying 
principles in multiple situations. Effective teaching practices accommodate effective ELS implementation 

6. An understanding of how the ELS is to be used and the expectations for learning must be clearly articulated. 
Jones (1990) observed that change is more manageable and occurs more easily when school districts articulate a 
common perception about what the change process entails. Once an ILS goes into a school building it is beyond 
the control of the designers and developers and it becomes the responsibility of the stakeholders in the change 
process to ensure that the ILS is properly implemented and that systemic change occurs 

Conclusions 



This study focused on describing the concerns and implementation practices of teachers implementing an 
ILS. Computer-delivered instruction and integrated learning systems represent one stage in a long succession of 
technologies and innovations embraced by education in the last several decades. Although the problem of 
implementing ILSs is relatively new, the issue of properly or adequately assimilating technology into the classroom 
and promoting its effective use has long been a matter of research and debate. Since the implementation of ILSs is a 
complex process, further studies that describe the implementation process for ILSs should be conducted. 

Implementation is better understood when contextual phenomena of the implementation are examined 
because the question of the instructional effectiveness of an ILS or other forms of computer-delivered instruction 
may be as much an implementation issue as a matter of instructional design. Ultimately, the potential of ILSs and 
ether forms of computer- delivered instruction may be unfulfilled when the technology is ineffectively or improperly 
implemented. 
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Computer-Delivered Instruction Configuration Matrix 
Integrated Learning System 



Rules for Rating: 

(1) Any information in the interview may be used to rate any single component 

(2) Rate to the highest level of use described by the respondent for any single component. 





5 

IDEAL USE 


4 3 

ACCEPTABLE USE 


2 

MINIMIMAL 

USE 


1 

UNACCEPTABLE 

USE 


Comoonent 1 
Instructional 
System and 
Design of ILS 


Describes 

individualized 

prescriptive 

strategies 


Describes 
instructional 
presentation and 
mastery of skills 


Describes 
enrollment levels 


Describes 
different content 
areas only 


Has no 

understanding of 
instructional design 
or no understanding 
is necessary 


Component 2 
Training in Use 
of ILS 


Initial training, 
continued training, 
and program updates 
are conducted 


Initial training and 
continued training is 
conducted 


Initial training or 
orientation is 
conducted 


Training is self- 
directed and 
occurs on-the- 
job 


Received no 
training 


Component 3 
On-Going 
Support System 
and 

Communication 
in Use of ILS 


Formal grade/ 
department level 
meetings to discuss 
ILS are conducted 


Building level 
meetings with vendor 
or principal to 
discuss ILS are 
conducted 


Technology 
committee meets 
periodically to 
discuss ILS 
instruction 


Informal 
discussions with 
lab manager or 
other teachers 


No attention is 
given to on-going 
support 


Component 4 
Instructional 
Goals or 
Expectations for 
Use of ILS 


Accomplishment of 
goals is celebrated 


Instructional goals 
for ILS are 
accomplished 


A plan for 
accomplishing 
instructional 
goals is stated 


Goals or 
expectations for 
ILS are stated 


No goals or 
expectations for 
ILS instruction are 
set 


Component 5 
Integration of 
ILS Courseware 
with Classroom 
Instruction 


ILS is used as a tool 

for regularly 

accomplishing 

classroom 

instructional 

objectives 


Plans lessons that 
integrate ILS 
courseware with 
classroom instruction 
in multiple subjects 
(Worksheets may be 
used) 


Plans lessons that 
integrate ILS 
courseware with 
classroom 
instruction in one 
subject 

(Worksheets may 
be used.) 


ILS courseware 
supplements 
classroom 
instruction 


ILS courseware is 
not integrated with 
classroom 
instruction 


Component 6 
Integration of 
ILS Courseware 
with Curriculum 


Sequence and 
selection of 
courses/lessons are 
adjusted to align with 
or support district 
curriculum 


ILS courseware 
supplements district 
curriculum in. 
multiple subjects 


ILS courseware 

supplements 

district 

curriculum in one 
subject 


ILS courseware 
is correlated to 
district 
curriculum 
when possible 


ILS courseware is 
not integrated with 
district curriculum 


Component 7 
Appropriate 
Selection of 
Courses, 
Enrollment 
Levels, and 
Options of ILS 


Individualized 
learning sequences 
are designed and 
modified based on 
test scores, 
monitoring student 
progress, forecasts of 
learning gains 


Learning sequences 
are individualized for 
each student based 
on test scores or 
monitoring of student 
progress 


Test scores or 
prior ILS 
performance are 
used to enroll 
students in same 
courses at 
different grade 
levels 


Students are 
enrolled in same 
courses at grade 
level 


Students are 
enrolled at 
beginning level of 
course or strand 


Component 8 
Effective 

Scheduling of ILS 


All students are 
scheduled for regular 
use and makeup 
sessions are provided 


All students are 
scheduled regular use 


Some students 
are scheduled for 
regular use 


Some students 
are scheduled 
for occasional, 
remedial or 
specialized use 


Students are not 
scheduled for either 
occasional or 
regular use 
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IDEAL USE 


4 3 

ACCEPTABLE USE 


2 

MINIMIMAL 

USE 


1 

UNACCEPTABLE 

USE 


ComDonent 9 


Orientation to rules 
and procedures is 
presented 


Rules and procedures 
are established and 
handed out to 
students in printed 
form 


Rules and 
procedures are 
established and 
posted in lab or 
classroom 


Some rules and 
procedures are 
established by 
the teacher 


No rules and 
procedures are 
established 


Clear Rules for 
Daily Procedures 
Using ILS 


Component 10 


Amount of 
instructional time is 
determined by 
targeted gain for 
students 


Students receive 
more than 30 minutes 
of ILS instruction per 
week and makeup 
sessions are provided 


Students receive 
more than 30 
minutes of ILS 
instruction per 
week 


Students receive 
at least 30 
minutes of ILS 
instruction per 
week 


Students do not 
receive regular 
weekly instruction 
using ILS 


Sufficient Time 
on Task for Each 
Student Using 
ILS 


Component 11 
Teacher 
Facilitation and 
Intervention 
Using ILS 


Continuously 
facilitates instruction; 
provides intervention 
strategies including 
worksheets, selected 
practice, tutoring, or 
small group 
instruction 


Continuously 
facilitates instruction 


Occasionally 
facilitates 
instruction or 
facilitate when 
students request 
assistance 


Facilitation and 
intervention Is 
provided 
primarily by lab 
manager 


Teacher is not 
present or does not 
facilitate ILS 
instruction 


Component 12 
Effective Use of 
Reinforcement 
and Motivational 
Strategies Using 
ILS 


Recognizes 
individual and group 
achievement through 
use of individual and 
group motivational 
strategies of 
motivational 
strategies involving 
parents or 

community sponsors 


Recognizes 

individual 

achievement through 
use of individual 
motivational 
strategies including 
certificates, wall 
charts, or individual 
competition 


Recognizes 

group 

achievement 
through use of 
group 

motivational 
strategies 
including 
contests or team 
activities 


Explains 
reasons for 
using ILS and 
encourages 
students to 
actively 
participate in 
ILS instruction 


No motivational 
strategies or 
activities are used 


Component 13 
Student Feedback 
and Use of 
Reports 

Generated by ILS 


Reports are used to 
provide information 
for determining 
classroom instruction* 
or classroom 
activities 


Reports are used to 
review student 
progress and modify 
student enrollment 


Reports are used 
for progress 
review by lab 
manager, teacher, 
or principal 


Reports are used 
infrequently or 
on a limited 
basis 


Reports are not run 
or distributed 


Component 14 
Instructional 
Assessment of 
ILS Courseware 


Evaluation or 
assessment of 
students includes 
mastery levels, lesson 
completion, or 
courseware content 
for multiple subjects 


Students receive a 
letter or numeric 
grade for ELS 
achievement in 
multiple subjects 


Students receive 
a letter or 
numeric grade 
for ILS 

achievement in 
one subject 


ILS is optional 
for inclusion in 
the evaluation 
or assessment of 
students 


ILS is not included 
in the evaluation or 
assessment of 
students 


Component 15 
Teacher 
Knowledge and 
Skills Using ILS 
Routines and 
Equipment 


Familiar with course 
content for multiple 
courses, can modify 
instructional levels or 
other student 
enrollment 
information, and can 
use custom reports or 
forecasting reports to 
make instructional 
decisions 


Familiar with course 
content for multiple 
courses and can 
modify instructional 
levels or other 
student enrollment 
information 


Familiar with 
course content 
and student 
resources for 
multiple courses 


Familiar with 
course content 
and student 
resources for 
one course 


No familiarity with 
course content, 
student resources, 
or management 
system 



Interview Guide for the Computer-Delivered Instruction Configuration Matrix 



Is it OK if we use the term ILS to stand for the integrated learning system including courseware, computers, and lab? 

1. Describe in your own words the organization of ILS instruction? 

2. How much formal training have you had in the use of ILS? What training have you received lately? 

3. What formal or informal communication such as meetings or discussions has occurred to support you in the use 
of ILS? (What do you do and who do you tell if you have a problem?) 

4. What goals or expectations do you set for your class for ILS instruction? How do you determine if these goals 
are accomplished? 

5. When planning for classroom instruction, how do you integrate or coordinate ILS instruction into classroom 
activities? 

6. Does ILS integrate with district or grade level curriculum? If so, in what ways and with what courses? 

7. How do you determine the courses, level, and sequence of instruction students receive using ILS? Are 
modifications to student enrollment ever made? If so, how do you determine what modifications are made? 

8. What students in your class receive ILS instruction? Are makeup classes provided for when students miss ILS 
instruction or when your class misses a scheduled lab time? 

9. Have rules or procedures been established for students using ILS? If so, how do students know these rules or 
procedures? 

10. How much time do students spend each day or week using ILS? How do you determine the amount of time 
students spend using ILS? 

1 1 . What do you actually do while the students are using ILS? 

12. How do you keep students motivated about using ILS? (Are there organized programs in your classroom or 
school to recognize student achievement using ILS?) 

13. How do you use the student reports generated by ILS? Is this information reported to the principal or lab 
manager? If so, how do’ they use the reports? 

14. Is ILS included in your evaluation and assessment of students? If so, in what ways do you use ILS for evaluation 
and assessment of students? 

15. What courses on the ILS are you most familiar with? What routines on ILS can your perform (student reports, 
custom reports, enrollment)? 
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